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Description 

BACKGROUND OF THE INVENTION 

5 [0001 ] The present invention relates to a system for managing audiovisual information, and in particular to a system 
for audiovisual information browsing, fOtering, searching, archiving, and personalization. 

[0002] Video cassette recorders (VCRs) may record video programs in response to pressing a record button or may 
be programmed to record video programs l>ased on the time of day. However, the viewer must program the VCR based 
on information from a television guide to identify relevant programs to record. After recording, the viewer scans through 

10 the entire video tape to select relevant portions of the program for viewing using the functionality provided by the VCR. 
such as fast fonward and fast reverse. Unfortunately, the searching arxl viewing is t>ased on a linear search, which may 
require significant time to locate the desired portions of the program(s) and fast forward to the desired portion of the 
tape. In addition, it is time consuming to program the VCR in light of the television guide to record desired programs. 
Also, unless the viewer recognizes the programs from the television guide as desirable it is unlikely that the viewer will 

75 select such programs to be recorded. 

[0003] RePlayTV and Ti Vo have developed hard disk b>ased systems that receive, record, and play television broad- 
casts in a manner similar to a VCR. The systems may be programmed with the viewer's viewing preferer>ces. The sys- 
tems use a telephone line interface to receive scheduling information similar to that available from a television guide- 
Based upon the system programming and the scheduling information, the system automatically records programs that 

20 may be of potential interest to the viewer. Unfortunately, viewing the recorded programs occurs in a linear manner and 
may require substantial time. In addition, each system must be programmed for an individual's preference, likely in a 
different manner. 

[0004] Freeman et aL, U.S. Patent No. 5,861 .881 , disclose an interactive computer system where subscribers can 
receive individualized content. 

25 [0005] With all the aforementioned systems, each individual viewer is required to program tiie device according to 
his particular viewing preferences. Unfortunately, each different type of device has different capabilities and limitations 
which limit the selections of the viewer. In addition, each device includes a different interface which the viewer may be 
unfamiliar with. Further, if the operator's manual is inadvertentiy misplaced it may be difficult for the viewer to efficiently 
program the device. 

30 

SUMMARY OF THE INVENTION 

[0006] The present invention overcomes tiie aforementioned drawbacks of the prior art by providing at least one 
description scheme. For audio and/or video programs a program description scheme provides information regarding 
35 the associated program. For the user a user description scheme provides information regarding the user's preferences. 
For the system a system description scheme provkles information regarding the system. The description schemes are 
independent of one another. In the preferred embodiment the system may use a combination of the description 
schemes to enhance its ability to search, filter, arxl browse audiovisual information in a personalized and effective man- 
ner. 

40 [0007] The foregoing arxJ other objectives, features and advantages of the invention will be more readily under- 
stood upon consideration of the following detailed description of the inverrtion. taken in conjunction with the accompa- 
nying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

45 

[0008] 

FIG 1 is an exerrplary embodiment of a program, a system, and a user, with associated description schemes, of 

an audiovisual system of the present invention. 
so FIG. 2 is an exemplary embodiment of the audiovisual system, including an analysis module, of FIG.1 . 

FIG. 3 is an exemplary embodiment of the analysis module of FIG. 2. 

FIG. 4 is an illustration of a thumbnail view (category) for the audiovisual system. 

FIG. 5 is an illusti^tion of a thumtxiail view (channel) for the audiovisual system. 

FIG. 6 is an illustration of a text view (channel) for the audiovisual system. 
55 FIG. 7 is an illustration of a frame view for the audiovisual system. 

FIG. 8 is an illustration of a shot view for the audiovisual system. 

FIG. 9 is an illustration of a key frame view the audiovisual system. 

FIG. 10 is an illustration of a highlight view for the audiovisual system. 
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FIG. 11 is an illustration of an event view for the audiovisual system. 

FIG. 12 is an illi^tration of a character/object view for the audiovisual system. 

FIG. 13 is an alternative embodiment of a program description scheme including a syntactic structure description 
scheme, a semantic structure description scheme, a visualization description scheme, and a m^a information 
5 description scheme. 

FIG. 14 is an exemplary embodiment of tiie visualization description scheme of FIG. 13. 

FIG. 15 is an exemplary emtxxiiment of the meta information description scheme of FIG. 13. 

FIG. 16 is an exemplary emljodiment of a segment description scheme for the syntactic structure description 

scheme of FIG. 13. 

70 FIG. 1 7 is an exemplary emtxxiiment of a region description scheme for the syntactic structure description scheme 
of FIG. 13. 

FIG. 18 is an exenplary emtxxiiment of a segment/region relation description scheme for the syntactic structure 
description scheme of FIG. 13. 

FIG. 1 9 is an exenrplary embodiment of an event description scheme for the semantic structure description scheme 
75 of FIG. 13- 

FIG. 20 is an exemplary en^Kxdtment of an object description scheme for the semantic structure description 
scheme of FIG. 13. 

FIG. 21 is an exemplary embodiment of an event/object relation graph description scheme for the syntactic struc- 
ture description scheme of FIG. 13. 

20 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0009] Many households today have many sources of audio and video information, such as multiple television sets, 
multiple VCR's, a home stereo, a home entertainment center, cable television, satellite television, internet broadcasts, 

25 world wide web. data services, specialized Internet services, portable radio devices, and a stereo in each of their vehi- 
cles. For each of tiiese devices, a different interface is normally used to obtain, select, record, and play the video and/or 
audio content For example, a VCR permits the selection of tiie recording times but the user has to correlate tiie televi- 
sion guide with the desired recording times. Another example is the user selecting a preferred set of preselected radio 
stations for his home stereo and also presumably selecting the same set of preselected stations for each of the user's 

30 vehicles. If another household member desires a different set of preselected stereo selections, the programming of 
each audio device would need to be reprogrammed at substantial inconvenience. 

[001 0] The present inventors came to the realization that users of visual information and listeners to audio informa- 
tion, such as for example radio, audio tapes, video tapes, movies, and news, desire to be entertained and informed in 
more than merely one uniform manner. In other words, tiie audiovisual information presented to a particular user should 

35 be in a format and include content suited to their particular viewing preferences. In addition, the format should be 
dependent on the content of the particular audiovisual information. The amount of infornnation presented to a user or a 
listener should be limited to only tiie amount of detail desired by the particular user at tiie particular time. For example 
with the ever increasing demands on the user's time, the user may desire to watch only 10 minutes of or merely the 
highlights of a baskett>al! game. In addition, the present inventors came to the realization tiiat the neoessity of program- 

40 ming multiple audio and visual devices with their particular viewing preferences is a txjrdensome task; especially when 
presented with unfamiliar recording devices when traveling. When traveling, users desire to easily configure unfamiliar 
devices, such as audiovisual devices in a hotel room, with their viewing and listening preferences in a efficient manner. 
[0011] The present inventors came to the further realization that a convenient technique of merely recording the 
desired audio and video information is not sufficient because the presentation of the information should be in a manner 

45 that is time efficient, especially in light of the limited time frequentiy available for the presentation of such information. 
In addition, tiie user should be able to access only that portion of all of the available information that the user is inter- 
ested in. while skipping the remainder of the information. 

[0012] A user is not capable of watching or otherwise listening to the vast potential amount of information available 
through all. or even a small portion of. the sources of audio and video information. In addition, with the increasing infor- 

50 mation potentially available, the user is not likely even aware of the potential content of Information that he may be inter- 
ested in. In light of the vast amount of audio, image, and video information, the present inventors came to the realization 
that a system that records and presents to the user audio and video information based upon the user's prior viewing 
and listenirig habits, preferences, and personal characteristics, generally referred to as user information, is desirat>le. 
In addition, the system may present such information leased on tiie capabilities of the system devices. This permits the 

55 system to record desirat»Ie information and to customize itself automatically to the user arxi/or listener. It is to be under- 
stood that user, viewer, and/or listener terms may be used interchangeability for any type of content Also, the user infor- 
mation should be portable between arxi usable by different devices so that otfier devices may likewise be configured 
automatically to the particular user's preferences upon receiving the viewing information. 
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[0013] In light of the foregoing realizations and motivations, the pres^ inventors analyzed a typical audio and 
video presentation environment to determine the significant portions of the typical audiovisual environment First; refer- 
ring to FIG. 1 the video, Image, and/or audio information 10 is provided or otherwise made availat^le to a user arKl/or a 
(device) system. Second, the video, image, and/or audio information is presented to the user from the system 12 

5 (device), such as a television set or a radio. Third, the user interacts both with the system (device) 1 2 to view the Infor- 
mation 10 in a desirable manner and has preferences to define which audio, image, and/or video information is obtained 
in accordance with the user information 14. After the proper identification of the different major aspects of an audiovis- 
ual system the present inventors then realized that information is needed to describe the informational content of each 
portion of the audiovisual system 1 6. 

10 [0014] With three portions of the audiovisual presentation system 1 6 identified, the functionality of each portion 
identified together with Its Interrelationship to the other portions. To define the necessary interrelationships, a set of 
description schemes containing data desaibing each portion is defined. The description schemes include data that is 
auxiliary to the programs 1 0, the system 12. and the user 1 4, to store a set of information, ranging from human readable 
text to encoded data, that can be used in enabling browsing, filtering, searching, archiving, and personalization. By pro- 

15 viding a separate description scheme describing the program(s) 10, the user 14, and the system 12, the three portions 
(program, user, and system) may be combined together to provide an interactivity not previously achievable. In addition, 
different programs 1 0, different users 1 4, and different systems 1 2 may be combined together in any combination, while 
still maintaining full compatibility and functionality. It is to be understood that the description scheme may contain the 
data itself or include links to the data, as desired. 

20 [0015] A program description scheme 18 related to to video, still image, and/or audio infornr^tion 10 preferably 
includes two sets of information, namely, program views and program profiles. The program views define logical struc- 
tures of the frames of a video that define how the video frames are potentially to be viewed suitable for efficient brows- 
ing. For example the program views may contain a set of fields that contain data for the identification of key frames, 
segment definitions between shots, highlight definitions, video summary definitions, different lengths of highlights. 

25 thumbnail set of frames, individual shots or scenes, representative frame of the video, grouping of different events, and 
a close-up view. The program view descriptions may contain thunt^nail. slide, key frame, highlights, and close-up views 
so that users can filter and search not only at the program level but also within a particular program. The description 
scheme also enables users to access information in varying detail amounts by supporting, for example, a key frame 
view as a part of a program view providing multiple levels of summary ranging from coarse to fine. The program profiles 

30 define distinctive characteristics of the content of the program, such as actors, stars, rating, director, release date, time 
stamps, keyword identification, trigger profile, still profile, event profile, character profile, object profile, color profile, tex- 
ture profile, shape profile, motion profile, and categories. The program profiles are especially suitable to facilitate filter- 
ing and searching of the audio and video information. The description scheme enables users to have the provision of 
discovering interesting programs that they may be unaware of by providing a user description scheme. The user 

35 description scheme provides information to a software agent that in turn performs a search arxJ filtering on behalf of the 
user by possibly using tiie system description scheme and the program description scheme information. It is to be 
understood that in one of the embodiments of the invention merely the program description scheme is included. 
[001 6] Program views contained in the program description scheme are a feature that supports a functionality such 
as close-up view. In tiie close-up view, a certain image object, e.g., a famous basketball player such as Michael Jordan. 

40 can be viewed up dose by playing back a close-up sequence that is separate from the original program. An alternative 
view can be incorporated in a straightforward manner. Character profile on the other hand may contain spatio-tenporal 
position and size of a rectangular region around the character of interest. This region can be enlarged by the presenta- 
tion engine, or the presentation engine may darken outside the region to focus the user's attention to the characters 
spanning a certain number of frames. Information within the program description scheme may contain data atx>ut the 

45 initial size or location of the region, movement of the region from one frame to another, and duration and ternns of the 
number of frames featuring the region. The character profile also provides provision for including text annotation and 
audio annotation about the chtaracter as well as web page information, and any other suitable information. Such char- 
acter profiles may include the audio annotation which is separate from and in addition to the associated audio track of 
the video. 

so [0017] The program description scheme may likewise contain similar information regarding audio (such as radio 
broadcasts) and images (such as analog or digital photographs or a frame of a video). 

[0018] The user description scheme 20 preferably includes the user's personal preferences, and information 
regarding the user's viewing history such as for example browsing history, filtering history, searching history, arxJ device 
setting Nstory. The user's personal preferences includes information regarding particular programs and categorizations 
55 of programs that the user prefers to view. The user description scheme may also include personal information about tiie 
particular user, such as demographic and geographic information. e.g. zip code and age. The explicit definition of the 
particular programs or attributes related tiiereto pemnits the system 16 to select those programs from the information 
contained within the available program description schemes 18 that may be of interest to the user. Frequentiy. the user 
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does not desire to learn to program the device nor desire to explicrtly program the device. In addition, the user descrip- 
tion scheme 20 may not be sufficiently robust to include explicit definitions describing all desirable programs for a par- 
ticular user. In such a case, the capability of the user description scheme 20 to adapt to the viewing habits of the user 
to accommodate different viewing characteristics not explicitly provided for or otherwise difficult to describe is ireful. In 

5 such a case, the user description scheme 20 may be augmented or any technique can be used to conpare the infor- 
mation contained in the user description scheme 20 to the available information contained in the program description 
scheme 18 to make selections. The user description scheme provides a technique for holding user preferences ranging 
from program categories to program views, as well as usage history. User description scheme information is persistent 
txjt can be updated by the user or by an intelligent software agent on behalf of the user at any arbitrary time. It may also 

70 be disabled by the user, at any time, if the user decides to do so. In addition, the user description scheme is modular 
and portable so that users can carry or port it from one device to another, such as with a handheld electronic device or 
smart card or transported over a network connecting multiple devices. When user descrption scheme is standardized 
among different manufacturers or products, user preferences become portable. For example, a user can personalize 
the television receiver in a hotel room permitting users to access information they prefer at any time and anywhere. In 

15 a sense, the user description scheme is persistent and timeless based. In addition, selected information within the pro- 
gram description scheme may be encrypted since at least part of the information may be deemed to be private (e.g., 
demographics). A user description scheme may be associated with an audiovisual program broadcast and corrpared 
vwth a particular user's description scheme of the receiver to readily determine whether or not the program's intended 
audience profile matches that of the user. It is to be urxierstood that in one of the embodiments of the invention merely 

20 the user description scheme is included, 

[0019] The system description scheme 22 preferably manages the Individ ualjDrograms and other data. The man- 
agement may Include maintaining lists of programs, categories, channels, users, videos, audio, and images. The man- 
agement may include the capabilities of a device for providing the audio, video, and/or images. Such capabilities may 
include, for exanple. screen size, stereo, AC3. DTS, color, black/white, etc. The management may also include relation- 

25 ships between any one or more of the user, the audio, and the images in relation to one or more of a program descrip- 
tion scheme(s) and a user description scheme(s). In a similar manner the management may include relationships 
between one or more of the program description scheme(s) and user description scheme(s). It Is to be understood that 
in one of the embodiments of the invention merely the system description scheme is included. 
[0020] The descriptors of the program description scheme and the user description scheme should overlap, at least 

30 partially, so that potential desirability of the program can be determined by corrparing descriptors representative of the 
same information. For exanpie. the program and user description scheme may include the same set of categories and 
actors. The program description scheme has no knowledge of the user description scheme, and vice versa, so that 
each description scheme is not dependant on the other for its existence. It is not necessary for the description schemes 
to be fully populated. It is also beneficial not to Include the pro-am description scheme with the user description 

35 scheme because there will likely be thousands of programs with associated description schemes which if combined 
with the user description scheme would resuH in a unnecessarily large user description scheme. It is desirable to main- 
tain the user description scheme small so that it is more readily portat)le. Accordingly, a system including only the pro- 
gram description scheme and the user description scheme would be beneficial. 

[0021] The user description scheme and the system description scheme should include at leas} partially overlap- 

40 ping fields. With overiapping fields the system can capture the desired information, which would otherwise not be rec- 
ognized as desirable. The system description scheme preferably includes a list of users and available programs. Based 
on the master list of available programs, and associated program description scheme, the system can match the 
desired programs. It is also beneficial not to include the system description scheme with the user description scheme 
because there will likely be thousands of programs stored in the system description schemes which if conrAjined with 

45 the user description scheme wouW result in a unnecessarily large user description scheme. It is desirable to maintain 
the user description scheme small so that it is more readily portable. For example, the user description scheme may 
include radio station preselected frequencies and/or types of stations, while the system description scheme includes 
the available stations for radio stations in particular cities. When traveling to a different city the user description scheme 
together with the system description scheme will permit reprogramming the radio stations. Accordingly, a system Includ- 

50 Ing only the system description scheme and the user description scheme would be beneficial. 

[0022] The program description scheme an6 the system description scheme should include at least partially over- 
lapping fields. With the overlapping fields, the system description scheme will be capable of storing the information con- 
tained vwthin the program description scheme, so that the information is properiy indexed. With proper indexing, the 
system Is capable of matching such information with the user information, if available, for obtaining and recording suit- 

55 able programs. If the program description scheme and the system description scheme were not overiapping then no 
information would be enacted from the programs and stored. System capabilities specified within the system descrip- 
tion scheme of a particular viewing system can be con-elated with a program description scheme to determine the views 
that can be supported by the viewing system. For instance, if the viewing device is not capat>le of playing back video, 
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its system description scheme may desaibe its viewing capabilities as limited to keyframe view and slide view only. Pro- 
gram description scheme of a particular program and system description scheme of the viewing system are utilized to 
present the appropriate views to the viewing system. Thus, a sewer of programs serves the appropriate views accord- 
ing to a particular viewing system's capabilities, which may be communicated over a network or communication channel 

5 connecting the sewer with user's viewing device. It is preferred to maintain the program description scheme separate 
from the system description scheme because the content providers repackage the content and description schemes in 
different styles, times, and formats. Preferably, the program description scheme is associated with the program, even if 
displayed at a different time. Accordingly, a system irx^uding only the system description scheme and the program 
description scheme would be beneficial. 

10 [0023] By preferably maintaining the independence of each of the three description schemes while having fields 
that correlate the same information, the programs 10, the users 14, and the system 12 may be interchanged with one 
another while maintaining the functionality of the entire system 16. Referring to FIG. 2. the audio, visual, or audiovisual 
program 38, is received by the system 16. The program 38 may originate at any suitable source, such as for example 
broadcast televrision, cable television, satellite television, digital television, Internet broadcasts, world wide web, digital 

15 video discs, still images, video cameras, laser discs, magnetic media, conputer hard drive, video tape, audio tape, data 
services, radio broadcasts, and microwave communications. The program description stream may originate from any 
suitable source, such as for exannple PSIP/DVB-SI information in digital television broadcasts, specialized digital tele\^- 
sion data services, specialized Internet services, world wide web, data files, data over the telephone, and memory, such 
as computer memory. The program, user, and/or system description scheme may be transported over a network (com- 

20 munication channel). For exanple, the system description scheme may be transported to the source to provide the 
source with views or other capabilities that the device is capable of using. In response, the source provides the device 
with image, audio, and/or video content customized or otherwise suitable for the particular device. The system 16 may 
include any device(s) suitable to receive any one or more of such programs 38. An audiovisual program analysis mod- 
ule 42 performs an analysis of the received pro-ams 38 to extract and provkie program related information (descrip- 

25 tors) to the description scheme (DS) generation module 44. The program related information may be extracted from the 
data stream including the program 38 or obtained from any other source, such as for example data transferred over a 
telephone line, data already transferred to the system 16 in the past or data from an associated file. The program 
related information preferatHy includes data defining both the program views arxi the program profiles available for the 
particular program 38. The analysis module 42 performs an analysis of the programs 38 using information obtained 

30 from (i) automatic audio-video analysis methods on the basis of low-level features that are extracted from the pro- 
gram(s). (ti) event detection techniques, (iii) data that is available (or extractable) from data sources or electronic pro- 
gram gukJes (EPGs. DVB-SI, and PSIP). and (iv) user information obtained from the user description scheme 20 to 
provkie data defining the program description scheme. 

[0024] The selection of a particular program analysis technique depends on the amount of readily available data 
35 and the user preferences. For example, if a user prefers to watch a 5 minute video highlight of a particular program, 
such as a baskettDall game, the analysis module 42 may invoke a knowledge based system 90 (FIG. 3) to determine the 
highlights that form the best 5 minute summary. The knowledge based system 90 may invoke a commercial filter 92 to 
remove commercials and a slow motion detector 54 to assist in creating the video summary. The analysis nrKXdule 42 
may also invoke other modules to bring information together (e.g., textual Infornr^tion) to author particular program 
40 views. For example, if the program 38 is a home video where there is no further information available then the analysis 
module 42 may create a key-frame summary by identifying key-frames of a mutti-level summary and passing to infor- 
mation to be used to generate the program views, and in particular a key frame view, to the description scheme. Refer- 
ring also to FIG. 3. the analysis module 42 may also include other sub-nxxiules, such as for example, a de-mux/decoder 
60, a data and service content analyzer 62. a text processing and text summary generator 64, a close caption analyzer 
45 66, a title frame generator 68, an analysis manager 70. an audiovisual analysis and feature extractor 72, an event 
detector 74. a key-frame summarizer 76. and a highlight summarizer 78. 

[0025] The generation module 44 receives the system informatk)n 46 for the system description scheme. The sys- 
tem information 46 preferably includes data for the system description scheme 22 generated by to generation module 
44. The generation module 44 also receives user information 48 including data for the user description scheme. The 

so user information 48 preferatHy includes data for the user description scheme generated within the generation nnodule 
44. The user input 48 may include, for example, meta information to be included in the program and system description 
scheme. The user description scheme (or corresponding information) is provided to the analysis module 42 for selective 
analysis of the program(s) 38- For exarrple. the user description scheme may be suitable for triggering the highlight 
generation functionality for a particular program and thus generating the preferred views arxJ storing associated data in 

55 to program description scheme. The generation module 44 and the analysis module 42 provide data to a data storage 
unit 50. The storage unit 50 may be any storage device, such as memory or magnetic media. 
[0026] A search, filtering, and browsing (SFB) module 52 implements to description scheme technique by parsing 
and extracting information contained within the description scheme. The SFB nxxjule 52 may perform filtering, search- 
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ing, and browsing erf the programs 38. on to basis of the information contained in the description schemes. An intelligent 
software agent is preferatrfy included within the SFB module 52 that gathers and provides user specific information to 
the generation module 44 to be used in authoring and updating the user description scheme (through the generation 
module 44). In this manner, desirable content may be provided to the user though a display 80. The selections of the 
5 desired program(s) to be retrieved, stored, and/or viewed may be programmed, at least in part, through a graphical user 
interface 82, The graphical user interface may also include or be connected to a presentation engine for presenting to 
information to to user through the graphical user interface. 

[0027] The intelligent management and consumption o1 audiovisual information using the multi-part description 
steam device provides a next-generation device suitable for the modem era of information overload. The device 
10 responds to changing lifestyles of individuals and families, and allows everyone to obtain the information they desire 
anytime and anywhere they want. 

[0028] An example of the use of the device may be as follows. A user comes home from work late Friday evening 
being happy the work week is finally over. The user desires to catch up with the events of the world and then watch 
ABC's 20/20 show later that evening. It is now 9 PM and the 20/20 show will start in an hour at 10 Pf^. The user is inter- 

15 ested in the sporting events of the week, and all the news about the Microsoft case with the Department of Justice. The 
user description scheme may include a profile indicating a desire that the particular user wants to obtain all available 
information regarding the Microsoft trial and selected sporting events for particular teams. In addition, the system 
description scheme and program description scheme provide information regarding the content of the available infor- 
mation that may selectively be obtained and recorded. The system, in an autonomous manner, periodically obtains and 

20 records the audiovisual Information that may be of interest to the user during the past week based on the three descrip- 
tion schemes. The device most likely has recorded more than one hour of audiovisual information so the irrformation 
needs to be condensed in some manner. The user starts interacting with the system with a pointer or voice commands 
to indicate a desire to view recorded sporting programs. On the display, the user is presented writh a list of recorded 
sporting events including Basketball and Soccer. Apparently the user's favorite Foott>all team did not play that week 

25 because it was not recorded. The user is interested in basketball games and indicates a desire to view games. A set of 
title frames is presented on the display that captures an important moment of each game. The user selects the Chicago 
Bulls game and indicates a desire to view a 5 minute highlight of the game. The system automatically generates high- 
lights. The highlights may be generated by audio or video analysis, or the program description scheme includes data 
indicating the frames that are presented for a 5 minute highlight. The system may hiave also recorded web-based textual 

30 information regarding the particular Chicago-Bulls game which may be selected by the user for viewing. If desired, the 
summarized information may be recorded onto a storage device, such as a DVD with a label. The stored information 
may also include an Index code so that it can be located at a later time. After viewing the sporting events the user may 
decide to read the news about the Microsoft trial. It Is now 9:50 PM and the user Is done viewing the news. In fact, the 
user has selected to delete all the recorded news items after viewing them. The user then remembers to do one last 

35 thing before 10 PM In the evening. The next day, the user desires to watch the VHS tape that he received from his 
brother that day. containing footage about his brother's new baby girl and his vacation to Peru last summer. The user 
wants to watch the whole 2-hour tape but he is anxious to see what the baby looks like and also the new stadium built 
in Lima, which was not there last time he visited Peru. The user plans to take a quick look at a visual summary of the 
tape, browse, and pertiaps watch a few segments for a couple of minutes, before the user takes bis daughter to her 

40 piano lesson at 10 /\M the next morning. The user plugs in the tape into his VCR, that is connected to the system, and 
invokes the summarization functionality of the system to scan the tape arxl prepare a summary The user can then view 
the summary the next nrorning to quickly discover the baby's looks, and playt)ack segments between the key-frames of 
the summary to catch a glimpse of the crying baby The system may also record the tape content onto the system hard 
drive (or storage device) so the video summary can be viewed quickly It is now 10:10 PM. ard it seems that the user 

45 is 1 0 minutes late for viewing 20/20. Fortunately, the system, based on the three description schemes, has already been 
recording 20/20 since 10 PM. Now the user can start watching the recorded portion of 20/20 as the recording of 20/20 
proceeds. The user will be done viewing 20/20 at 1 1 :10PM. 

[0(^9] The average consumer has an ever increasing number of multimedia devices, such as a home audio sys- 
tem, a car stereo, several home television sets, web browsers, etc. The user currently has to customize each of the 

so devices for optimal viewing and/or listening preferences. By storing the user preferences on a removable storage 
device, such as a smart card, the user may insert the card including the user preferences irrto such media devices for 
automatic customization. This results in the desired programs being automatically recorded on the VCR, and setting of 
the radio stations for the car stereo and home audio system. In this manner the user only has to specify his preferences 
at most once, on a single device and subsequently, the descriptors are automatically uploaded Into devices by the 

55 removat>le storage device. The user desaiption scheme may also be loaded into other devices using a wired or wire- 
less network connection, e.g. that of a home network. Alternatively, the system can store the user rtstory and aeate 
entries In the user description scheme based on the's audio and video viewing habits. In this manner, the user would 
never need to program the viewing information to obtain desired Information. In a sense, the us&r d^riptor scheme 
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enables modeling of the user by providing a central storage for the user's listening, viewing, browsing preferences, and 
user's behavior. This enables devices to be quickly personalized, and enables other components, such as intelligent 
agents, to communicate on the basis of a standardized description format, and to make smart Inferences regarding the 
user's preferences. 

5 [0030] Many different realizations and applications can be readily derived from FIGS. 2 and 3 by appropriately 
organizing and utilizing their different parts, or by adding peripherals and extensions as needed. In its most general 
form, FIG. 2 depicts an audiovisual searching, filtering, browsing, and/or recording appliance that Is personalizable. The 
list of more specific applications/implementations given betow is not exhaustive but covers a range. 
[0031] The user description scheme is a nr^ajor enabler for personalizable audiovisual appliances. If the structure 

10 (syntax and semantics) of the description schemes is known amongst multiple appliances, the user (user) can carry (or 
otherwise transfer) the information contained within his user description scheme from one appliance to another, per- 
haps via a smart card-where these appliances support smart card interface- in order to personalize them. Personali- 
zation can range from device settings, such as cfisplay contrast and volume control, to settings of television channels, 
radio stations, web stations, web sites, geographic information, and demographic information such as age, zip code etc. 

15 Appliances that can be personalized may access content from different sources. They may be connected to the web, 
terrestrial or cable broadcast, etc., and they may also access multiple or different types of single media such as video, 
music, etc. 

[0032] For example, one can personalize the car stereo using a smart card plugged out of the home system and 
plugged into the car stereo system to be able to tune to favorite stations at certain times. As another example, one can 

20 also personalize television viewing, for example, by plugging the smart card into a remote control that in turn will auton- 
orriously command the television receiving system to present the user information about current and fixture programs 
that fits the user's preferences. Different members of the household can instantly personalize the viewing experience 
by inserting their own smart card into the family remote. In the absence of such a remote, this same type of personali- 
zation can be achieved by plugging in the smart card directiy to the television system. The remote may likewise control 

25 audio systems. In another implementation, the television receiving system holds user description schemes for multiple 
users (users) in local storage and identify different users (or group of users) by using an appropriate input interface. For 
exanple an interface using user-voice identification technology. It is noted that in a networked system the user descrip- 
tion scheme may be transported over the network. 

[0033] The user description scheme is generated by direct user input, and by using a software tiiat watches the 
30 user to determine his/her usage pattern and usage history. User description scheme can be updated in a dynamic fash- 
Ion by the user or automatically A well defined and structured description scheme design allows different devices to 
interoperate witii each other. A modular design also provides portability. 

[0034] The desaiption scheme adds new functionality to tiiose of the current VCR. An advanced VCR system can 
learn from the user via direct input of preferences, or by watching the usage pattern and history of the user. The user 

35 description scheme holds user's preferences users and usage history. An intelligent agent can then consult with the 
user description scheme and obtain information that it needs for acting on behalf of the user. Through the intelligent 
agent, the system acts on behalf of the user to discover programs that fit the taste of the user, alert the user about such 
programs, and/or record them autonomously. An agent can also manage the storage in the system according to the 
user description scheme, Le.. prioritizing the deletion of programs (or alerting the user for transfer 4p a removable 

40 media), or determining tiieir compression factor (which directly impacts their visual quality) according to user's prefer- 
ences and history 

[0035] The program description scheme and the system description scheme work in collaboration with the user 
description scheme in achieving some tasks. In addition, the program description scheme and system description 
scheme in an advanced VCR or other system will enat^e the user to browse, search, and filter audiovisual programs. 

45 Browsing in the system offers capabilities that are well beyond fast forwarding and rewinding. For instance, the user can 
view a thumbnail view of different categories of programs stored in the system. The user then may choose frame view, 
shot view, key frame view, or highlight view, depending on their availability and user's preference. These views can be 
readily invoked using the relevant Information in the program description scheme, especially in program views. The user 
at any time can start viewing the program either in parts, or in its entirety. 

50 [0036] In tills application, the program description scheme may be readily available from many services such as: (i) 
from broadcast (carried by EPG defined as apart of ATSC-PSIP (ATSC-Program Service Integration Protocol) in USA 
or DVB-SI (Digital Video Broadcast-Service Information) in Europe); (ii) from specialized data services (in addition to 
PSIP/DVB-SI); (iii) from specialized web sites; (iv) from the media storage unit containing the audiovisual content (e.g.. 
DVD): (v) from advanced cameras (disci^ed later), and/or may be generated (i.e., for programs that are being stored) 

55 by the analysis rrxxiule 42 or by user input 48. 

[0037] Contents of digital still and video cameras can be stored arKt managed by a system that irrplements the 
description schemes. e.g.. a system as shown In FIG. 2. Advanced cameras can store a program description scheme, 
for instance, in addition to the audiovisual content itself. The program description scheme can be generated either in 
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part or in its entirety on the camera itself via an appropriate user Input interface (e.g.. speech, visual menu drive, etc.). 
Users can input to the camera the program description scheme information, especially those high-level (or semantic) 
information that may otherwise be difficult to automatically extract by the system. Some camera settings and parame- 
ters (e.g., date and time), as well as quantities computed in the camera (e.g., color histogram to be included in the color 

5 profile), can also be used in generating the program description scheme. Once the camera is connected, the system 
can browse the camera content, or transfer the camera content and its description scheme to the local storage for future 
use. It is also possitjle to update or add information to the description scheme generated in the camera 
[0038] The IEEE 1394 and Havi standard specifications enable this type of "audiovisual content" centric communi- 
cation among devices. The description scheme API's can be used in the context of Havi to browse and/or search the 

10 contents of a camera or a DVD which also contain a description scheme associated with their content, i.e.. doing more 
than merely invoking the PLAY API to play back and linearly view the media. 

[0039] The description schemes may be used in archiving audiovisual programs in a database. The search engine 
uses the information contained in the program description scheme to retrieve programs on the basis of their content TTie 
program description scheme can also be used in navigating through the contents of the datatjase or the query results. 
15 The user description scheme can be used in prioritizing the results of the user query during presentation. It is possible 
of course to make the program description scheme more conrprehensive depending on the nature of the particular 
application. 

[0040] The description scheme fulfills the user's desire to have applications that pay attention and are responsive 
to their viewing and usage habits, preferences, and personal demographics. The proposed user description scheme 
20 directly addresses this desire in its selection of fields and interrelationship to other description schemes. Because the 
description schemes are modular in nature, the user can port his user descriptioi^scheme from one device to another 
in order to "personalize* the device. 

[0041 ] The proposed description schemes can be incorporated into current products similar to those from TiVo and 
Replay TV in order to extend their entertainment informational value. In particular, the description scheme will enable 

25 audiovisual browsing and searching of programs and enable filtering within a particular program by supporting multiple 
program views such as the highlight view. In addition, the description scheme will handle progranns coming from 
sources other than television broadcasts for which TiVo and Replay TV are not designed to handle. In addition, by 
starxiardization of TiVo and Replay TV type of devices, other products may be interconnected to such devices to extend 
their capabilities, such as devices supporting an MPEG 7 description. MPEG-7 is the Moving Pictures Experts Group - 

30 7, acting to standardize descriptions and description schemes for audiovisual information. The device may also be 
extended to be personalized by multiple users, as desired. 

[0042] Because the description scheme is defined, the intelligent software agent can communicate among them- 
selves to make intelligent inferences regarding the user's preferences. In addition, the development and upgrade of 
intelligent software agents for browsing and filtering applications can be simplified based on the standardized user 
35 desaiption scheme. 

[0043] The description scheme is multi-nx)dal in the following sense that it holds both high level (semantic) and low 
level features and/or descriptors. For example, the high and low level desaiptors are actor name and motion nrxxlel 
parameters, respectively. High level descriptors are easily readat»le by humans while low level descriptors are more 
easily read by machines and less understandable by humans. The program description scheme caabe readily harmo- 

40 nized with existing EPG. PSIP. and DVB-SI information facilitating search and filtering of broadcast programs. Existing 
services can be extended in the future by incorporating additional information using the compliant description scheme. 
[0044] For exannple, one case may include audiovisual programs that are prerecorded on a media such as a digital 
video disc where the digital video disc also contains a description scheme that has the same syrrtax and semantics of 
the description scheme that the FSB module uses. If the FSB module uses a different description scheme, a transcoder 

45 (converter) of the description scheme may be employed. The user may want to browse and view the content of the dig- 
ital video disc. In this case, the user may not need to invoke the analysis nrxxJule to author a program description. How- 
ever, the user may want to invoke his or her user description scheme in filtering, searching and browsing the digital 
video disc content. Other sources of program information may likewise be used in the same manner. 
[0045] It is to be understood tfiat any of the techniques described herein with relation to video are equally applicable 

so to images (such as still image or a frame of a video) and audo (such as radio). 

[0046] An example of an audiovisual interface is shown in FIGS. 4-1 2 which is suitable for the preferred audiovisual 
desaiption scheme. Referring to FIG. 4. by selecting the thumbnail function as a function of category provides a display 
witii a set of categories on the left hand side. Selecting a particular category, such as news, provides a set of thumbnail 
views of different programs tfiat are cunrently available for viewing. In addition, the different programs may also include 

55 programs that will be available at a different time for viewing. The tfiumbnail views are short video segments that pro- 
vide an indication of the content of the respective actual program that it corresponds with. Referring to FIG. 5. a thumb- 
nail view of available programs in terms of channels may be displayed, if desired. Refening to FIG. 6. a text view of 
available programs in terms of channels may be displayed, if desired. Referring to FIG. 7. a franDe view of particular pro- 
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grams may be displayed, rf desired. A representative frame is displayed in the center of the display with a set of repre- 
sentative frames of different programs in the left hand column. The frequency of the number of frames may be selected, 
as desired. Also a set of frames are displayed on the lower portion of the display representative of different frames dur- 
ing the particular selected program. Referring to FIG. 8. a shot viev/ of particular programs may be displayed, as 

5 desired. A representative frame of a shot is displayed in the center of the display with a set of representative frames of 
different programs in the left band column. Also a set of shots are displayed on the lower portion of ^e display repre- 
sentative of different shots (segments of a program, typically sequential in nature) during the particular selected pro- 
gram. Referring to FIG. 9, a key frame view of particular programs may be displayed, as desired. A representative frame 
is displayed in the center of the display with a set of representative frames of different programs in the left hand column. 

10 Also a set of key frame views are displayed on the lower portion of the display representative of different key frame por- 
tions during the particular selected program. The number of key frames in each key frame view can be adjusted by 
selecting the level. Referring to FIG. 10, a highlight view may likewise be displayed, as desired. Referring to FIG. 11 , an 
event view may likewise be displayed, as desired. Referring to FIG. 12, a character/object view n^y likewise be dis- 
played, as desired. 

15 [0047] An example of the description schemes is shown below in XML The description scheme may be imple- 
mented in any language and include any of the included descriptions (or more), as desired. 

[0048] The proF>osed program description scheme includes three major sections for describing a video program. 
The first section identifies the described program. The second section defines a number of views which may be useful 
in browsing applications. The third section defines a nun^er of profiles which may be useful in filtering and search appli- 
20 cations. Therefore, the overall structure of the proposed description scheme is as follows: 

<?XML veralon-^1.0-'> 

25 



30 



35 



40 



45 



SO 



55 
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<!DOCTYVK MPEG-7 STSTEM *tnpeg-7 . dtd^> 
<Pr ograml denci ty> 

<PrograiiiID> ... </PrograttID> 
<PrDgrainNaiiie> ... </Pro9raiiiNaiiie> 
<SourceLocation> . . . </SourceLocatjLOii> 
</ Programldenti ty> 
<PrograinVicw3> 

<Thu2Dbnailview> ... </TiiumbnailView> 
<SXideView> ... </SiideView> 
<FrameView> ... </FraxiieViftw> 
<ShotView> ... </ShotVlew> ' 
<KeyrrameView> </KeyrraiBeView> 
<Hi9hIighCView> ... </HighXightView> 
<EventVi€vr> ... </£ventview> 
<Clo3«0pView> ... </CloseOpView> 
<AXternaceView> ... </AltecnateVlew> 
-< / Pr ogramVi ewa> 
<PrograiiiProfxlc3> 

<GeneraXProfile> ... </G«neralProflle> 

<CategoryProfile> </CategoryProf ile> 

<OateTimeProfile> ... </DaterijneProfile> . 
<KeywordProfiJ.e> </]CeywordProflIe> 
<TriggerProfilc> ... </TriggerProfile> 
<StillProfile> ... </StlXlProf il«> 
<EventProfile> </Cv«ntProflle> 
<CharacterProfiIe> . • . </CharacterProflXe> 
<ObjectProtile> </Qbj<ct:Pro£iX«> 
<CoXorProfiX«> . . . </CoXorPxof iX«> 
<Textur€Proflle> ... </Texrt2MProflXe> 
<Shap«ProfiX«> . . . </Shap«ProriXe> 
<MotlonProf iX«> . . , </MotionProtlX«> 
</ProgramProfiX«3> 
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Program Identity 
[0049] 
5 • Program ID 

< Prog rami 0> program-id </Prograi&ID> 

10 

The descriptor ( ProgramID ) contains a number or a string to identify a program. 
Program name 

15 <ProgramNaxne> program- name </PrograAName> 



The descriptor ( ProgramName ) specifies the name of a program. 

20 

Source location ^ 

<SourceI^cation> sourc«-ttrl </SoarceLocatioii> 

25 

The descriptor ( SourceLocation > specifies the location of a program in URL forrrat 
Program Views 
30 [0050] 

ThumtHiail view 

<Thumbnai lView> 

35 

<Xroage> thximbnail- image </Ifflage> 
< /Thumbna ilView> 

40 

The descriptor ( ThumbnailView > specifies an image as the thumbnail representation of a program. 
Slide view 

45 <SXi.deView> £ra&e-ld ... </SlideView> 

The descriptor ( SlideView) spedfies a number of frames in a program which may be viewed as snapsix)ts or 
in a slide show manner. 

so 

Frame view 

<FmeVlevr> atert-frane-id «x&d-f raaw-id </FraaeView> 

55 

The descriptor ( Frame View) specifies the start and end frames of a program. This is the most basic view of a 
program and any program has a frame view. 
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Shot view 

<ShotView> 

<5hoc id«'*''> acart-fraae-id cnd-frame-ld dl^lay- frame- id </Shot> 
<Shot id*"*^ start -frame- id «nd- frame- id di^Iay-f rame— id </Shot> 

</ShotView> 



The descriptor ( ShotView) spec'rfies a number of shots in a program. The < Shot) descriptor defines the start 
arxJ end frames of a shot. It may also specify a frame to r^resent the shot. 

Key-frame view 

<KeyFrameView> 

<KeyFrame3 level-''*'> 

<Clip id-**^ 3tart-frame-id end- frame-id display-frame-id </Clip> 
<Clip id«**> 3tart-framc-id end-frame-id display-frame-id </Clip> 

</KeyFrames> 

<KeyFrames level-**'> 

<Clip id-**> atart-frame-rid end-frame-id display-f rame*id </Cllp> 

<Clip id«**> start-frame-id end- frame-id display- frame-id </Clip> 

</KeyFrames> 
• « • 

</ReyFraffieView> 



The descriptor < KeyFrameView > specifies key frames in a program. The key frames may be organized in a 
hierarchical manner and the hierarchy is captured by the desaiptor < Key Frames) with a level attribute. The clips 
which are associated with each key frame are defined by the descriptor < CRp) . Here the display frame in each clip 
is the corresponding key frame. 
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<HighlightView> 

<Highlight length-* 

<Clip !<!•**'> start -frame-id end-frame- id displays frame -id </Clip> 
<Clip id-'""> start -frame- id end-frame-id display- frame-id </Clip> 

</Highlight> 

<Hi9hligbt length-*** > 

<Clip id-''-> start-frame- id end-frame- id display- frame-id </Clip> 
<Clip id-*''> start- frame-id end-f raxae-id display- frame-id </Clip>' 

</Highlight> 

</HighlightVi«w> 



The descriptor ( Highlight View) specifies dips to form highlights of a program. A program may have different 
versions of highlights which are tailored into various time length. The dips are grouped into each version of high- 
light which is specified by the desaiptor < Highlight) with a length attribute. 

Event view 



<EventView> 

<£vent:s name— **'> 

<Clip id-**^ start-frame-id end-frame-id display-frame-id </Clip> 
<Clip ld-^*> start- frame-id end-frame-id di^lay-f rame-id </Clip> 

</Events> 
<Sventui name-**'> 

<Clip id-**> start- frame-id end-frame-id display-frame-id </Clip> 
<Clip id-**> ^tart-frame-id end-frame-id display-framc-id </Clip> 

</Event3> 
</EventView> 
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The descriptor (EverttView) specifies clips which are related to certain events in a program. The clips are 
grouped into the corresponding events which are specified by the descriptor < Event) with a nanne attribute. 

Close-up view 

<Cloaet7pVittw> 

<Target tiaiDe-*'*> 

<Clip id~**> start- frame- id end-frane-id display- frame-id </Clip> 
<Clip id-^*> start-frame-id end-fraae-id displa7-frame-id </Clip> 
♦ » • 
</Target> 

CTarget aaiae«**'> • 

<Clip id-*''> start-fraM-id end-frame-id display-iTraiae-id </Clip> 
<Clip id-***^ start-frame-id cnd-framo-id display-£raxne-id </Clip> 

</Target> 

</Clo3eOpView> 

The descriptor (CloseUpView) specifies clips which may be zoomed in to certain targets in a program. The 
clips are grouped into the con-esponding targets which are specified by the descriptor (Target) with a name 
attribute. 

Alternate view 

<Alte£nateView> 

<AltemateSource id-*'*> source-url </AltemateSource> 
<AltemateSource id-**> source-url </AltemateSource> 

</AltemateView> 

The descriptor ( Alternate View) specifies sources which may be shown as alternate views of a program. Each 
alternate view is specified by the descriptor < AlternateSource ) with an id attribute. The locate of the source may 
be specified in URL format. 
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Program Profiles 
[0051] 
5 * General profile 

10 



15 



20 



25 



<G«ncralProfile> 

<Ticle> title-text </Tltle> 
<Ab3tract> abstract-text </Ab3tract> 
<Audlo> volce-aanotation < /Audio 
<Www> web-pa ge~arl </Www> 
<ClosedCaptioB> yea /no </Clo3edCaption> 
<X*an9ua9e> language-^nane </Langtia9e> 
<Rating> rating </Rating> 
<Lengtti> time </Length> 
<Author3> author-name . . , </Autbor3> 
<Producer3> producer*name • . . </ Producers > 
<Director3> director-naxae - - . </Di rectors > 
<Actor3> actor-xiame . . . </Actors> 



30 



</GeneralProfiIe> 



35 



40 



45 



SO 



55 



The descriptor (GeneralProfile) desaibes the general aspects of a program. 
Category profile 

<CategoryProf ile> category-name . • . </CategoryProf ile> 

The descriptor ( Categor/Profile > specifies the categories under which a program may be classified. 
Date-time profile 

XDateTiinePro£ile> 

<ProductlonDate> date </ProductionOate> 
<ReleaseOate> date </ReIeaseOate> 
<RecordlngDate> date </KecordingDate> 
<KecordixxgTi2na> tine </RecordingTi»e> 

</DateTimeProfiXtt> 
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The descriptor < DateTimeProfile) specifies various date and time information of a program. 
Keyword profile 

5 

<KeywordPrcf ile> keyword - . . </KeywordProf ile> 

^0 The descriptor < KeywordProfile) specifies a number of keywords which may be used to filter or search a pro- 

gram. 

TVIgger prof le 

15 <TriggerProf JJ.e> trigger-f rame-id . . . </TriggerFro£ile> 

The descriptor < TriggerProf lie > specifies a number of frames in a program which may be used to trigger certain 
20 actions while the playt>ack of the program. 

25 
30 
35 
40 
45 
50 
55 
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Still profile 



10 



15 



20 



25 



30 



35 



40 



45 



SO 



<StillProfile> 

<5till iel-^-'> 

<HotKegion id 

<Location> xl yl x2 y2 </Locatlon> 
<T«xt> text-annotation </Text> 
<Audio> Toice-annotation </Audio> 
<Www> web-page-url </Www> 

</HotIlegion> 

<HotRcgion id 

<I^cation> xl yl x2 y2 </Iocatioa> 
<Text> text -anno tat ion </Tcxt> 
<Aiidlo> voice-annotation </Audio> 
<Www> web-page-url </l«ww> 

</HotRegion> 

</Still> 
<Still id-''''> 

<HotRegion id 

<Location> xl yl x2 y2 </Location> 
<Text> text-annotation </Text> 
<Audio> voice-annotation </Audio> 
<Www> web-page-url </Www> 

</HotRegion> 

<HotRegion id 

<Location> xl yl x2 yZ </Location> 
<Text> text-annotation </Text> 
<Audio> voice*annotation <</Audio> 
-<Www> mb-p«ge-urX </Www> 

</HotRegion> 



55 
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</Still> 



</StillProfile> 



The descriptor ( StillProf Me ) specifies hot regions or regions of interest within a frame. The frame is specified 
by the descriptor < Still) with an id attribute which corresponds to the frame-Id. Within a frame, each hot region is 
specified by the descriptor < HotRegion) with an id attritHJte. 

Event profile 



<EventProfile> _ 

<EventLiat> event-name </EvcntlJ.3t> 
<Event name— ^'*> 

<Wvrw> ireb-page-url </Www> 
<Occarrence id— 

<Ducatlon> 3 tart- frame- id end-frame -id </Dur^ation> 
<Text> text-annotation </Text> 
<]Vudio> voice- annotation </Audio> 
</Occttrrence> 
<Occiirrence id-*"> 

<Duration> « tart- frame-id end- frame- id </Duration> 
<Te3et> text-annotation </Text> 
<Audio> voice-annotation </Audio> 
</Occuxrence> 
* • • 
</Event> 
<Event name-*'*^ 
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<Www> w«b-page-url </Www> 
<Occurrciice Ld^"> 

<Oiiratioii> ataxt-frame-id «nd- frame-id </Durat:lon> 

<Taxt> tcxt-annotarion </Text> 

<Audio> voice-annotation </AQdio> 
</Occurr«nce> 
<Occurrence id«*'*'> 

<Ouratlon> start -frame-id end- frame-id </Duracion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Attdio> 
</Occtarrence> 

</Event> 

</E vent Prof ile> 



The descriptor < EventProfile) specifies the detailed information for certain events in a program. Each event ts 
specified by the descriptor < Event) with a name attribute. Each occurrence of an event is specified by the descrip- 
tor (Occurrence) with an id attribute which may be matched with a dip id under ( EventView). 

Character profile 



<QiaracterProfile> 

<Characteria.st> character-name </CharacterLi3t> 
<Character naine-**> 

<ActorNaifte> actor-name </Actort«am«> 

<6ender> aale </GBndmx> 
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<A9tt> ag« </Ag«> 

<ttvrM> web-page-uxl </Wwt$> 

<Occurrence id-**> 

<Ouracxon> gtart* CrniM id «nd-rraiae-id </Duraclon> 

<Location> frame: Ixl yl x2 y21 </Location> 

<Motioxx> V. V, V, V. V, </Motion> 

<Text> text -annotation </T«xt> 

<Audio> voice-dxuxotation </Audio> 
</Occurr«nce> 
<Occarrenci» id«*'*> 

<D\aration> st:art*£raiDe-id end-frame-id </Duration> 

<Location> fcaxna: [xl yl x2 y2) ... </Location> 

<Motion> V. Ty V, V. V, </Motion> • 

<Text> text -annotation </Text> 

<Audio> voice-annotation </Audio> 
< /Occur rence> 

</C3iaracter> 
<Cbaracter naxoe— 

<Act:orHaiiie> actor-name </ActorNai&e> 

<Gender> male </Gender> 

<A9e> age </A9e> 

<»ww> web-page-url </www> 

<Occarrence id-'**'> 

<Dura^on> start-traine-id «zul*£raoie*id </Daratioa> 

<Location> frame: latl yl ac2 y2J ... </Locatlon> 

<Motioxi> V, V, V, </Motion> 

<Text> rext-annotation </T«t> 

<Audio> volce*annotation </ftudio> 
</Occurr«nce> 
<Occurrence id«»''''> 

<Ouratlon> srarr-fraae-id end-frane-id </OaraUon> 
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<X«ocatioa> frana: (xl yX 3c2 y2] ... </Ioca^loa> 
<Motioxi> v» V, </MotJLcm> 

<Text:> text-annotation </Text> 
<Aadio> voice-annotation </Aiiciio> 
< /Occur rcnce> 

</Character> 

</CharacterProfiie> 



The descriptor ( CharacterProfile) specifies the detailed information for certain characters in a program. Each 
character is specified by the descriptor < Character) with a name attribute. Each occurrence of a character is spec- 
ified by the descriptor (Occurrence) with an id attribute which may be matched with a clip id under 
(CloseUpView). 

Object profile 

<ObjectPj:ofile> 

<ObjectXJ.3t> otoject-name — </0bjectI*i3t:> 
<Objec^ nane-'*"^ 

<Www> web-page-url </Www> 
<Occurrenc« ld"**> 

<Oi2ratlon> staxt-fraawid «nd-£raiBe*id </Ouration> . 
<Location> frasM: Cxi yl x2 y2J </Xocation> 
<Motion> V. V, -v. </Motlon> 
<Text> tex^-annotation </Text> 
<Audio> ▼oice*annotation </Audio> 
< /Occurrenee> 
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<Occorrttzxce id»**> 

<Duration> 3tart-frame-id end-frame-id </Duxation> 

<Locatioii> frame :fxl yl x2 y2] </Location> 

<Hotion> V. Vy V, V. </Hoci<m> 

<Text> cext -anno tat ion </Text> 

<Audlo> voice-annotation </Audlo> 
</Occurrence> 

</Object> 
<ObJect najne-'"> 

<Www> web-page-ucl </Www> 
<Occurrence id-'*''> 

<Duration> 3tart-f rame-id end-frame-id </Duration> 

<Locacion> frame :(xl yl x2 yl] </Location> 

<Motion> v. v, </Motion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
< /Occur rence> 
<Occurrence id-'**> 

<I>uration> atart-frame-id end-frame-id </Duration> 

<I^cation> frame: [xl yl x2 y2] </Locatlon> 

<Motion> V. Vy V. v. </Motion> 

<Text> t«xt-annotation </Text> 

<Audio> voice-annotation </Audio> 
< /Occorr ence> 

</ObJ«ct> 

</(») j ectProfilO 



The descriptor < ObjectProfile) specifies the detailed information for certain objects in a program. Each object 
55 Is specified by the descriptor (Object) with a name attribute. Each occurrence of a object is specified by the 
descriptor < Occurrence > with an id attribute which may be matched with a clip id under < CloseUpView ) . 
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Color profile 

<ColorProfile> 
</ColorProfile> 



10 The desaiptor ( ColorProf ile ) specifies the detailed color information of a program. All MPEG-7 color descrip- 

tors nrtay be placed under here. 

Texture prof He 

75 <TexturePro£xIe> 

</Textxir«Prorile> 

20 

The descriptor < TextureProf ile > specifies the detailed texture infornnation of a program. All MPEG-7 texture 
descriptors may be placed under here. 

25 • Shape profile 

<SKa^Pro £lle> 
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45 
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</ShapeProfile> 



The descriptor <ShapeProfile> specifies the detailed shape information of a program. All MPEG-7 shape 
35 descriptors may be placed under here. 

Motion profile 

<MotionProfile> 
</MotionPro£ile> 



The descriptor < MotionProfile ) specifies the detailed motion information of a program. All MPEG-7 motion 
descriptors may be placed under here. 

User Description Scheme 

[0052] The proposed user description scheme includes three major sections for describing a user. The first section 
identifies the described user. The second section records a number of settings which may be preferred by the user. The 
third section records some statistics which may reflect certain usage patterns of the user. Therefore, the overall struc- 
ture of the proposed description scheme is as follows: 
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<?XML ver3ion-^l,0'> 

<!OOCTYPE MPEG-7 SYSTEM *ii^>eg-7 .dtd*> 
<03erldentity> 

<03erID> . . . </UderID> 

<UserHame> </D3eniazaft> 

</OaerIdcnt:ity> 
<UserPreferences> 

<Brow3ln9Preferences> ... < /Brows in9Pref«rence9> 

<Filterin9Preference3> </FilteringPreference5> 

<SearchPref erences> . . . </SearchFre£erence3> 

<DevicePreference3> - . . </ Device Preference5> 
</U3e rPre f erences> 
<03erHi3tory> 

<Brow3ingHistory> • </Brow3ingHi3tory> 

<FilteringHi5tory> ... </FilteringHistory> 

<S«archHistory> </SearchHistory> 

<DcviceHistory> . . . </0€viceHi3tory> 
</03erHistox:y> 
<03erD€mographiC5> 

<Age> </Age> 

<Gender> . - - </G€nder> 

<ZIP> </ZIP> 
</UserOeittographXcs> 



User Identity 

45 

[0053] 

User ID 

so <OserID> user-id </0serIO> 



The descriptor < UserlD ) contains a number or a string to identify a user. 

55 
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User name 

<XJserH^mm> user-name </03erName> 

5 

The descriptor ( UserName ) specifies the name of a user. 
User Preferences 

10 

[0054] 

Browsing preferences 

15 

<B rows mgPre f e rences> 
<View3> 

•cViewCategory i.d-*'*'> view- id . , . </ViewCategory> 
<VicwCategory id-^*> vie**- id . . . </ViewCategory> 

</Viewa> 

<rrameFre<iuency> frequency • * .<FraioeFretiaency> 
<Shotrre<iuency> frequency ... <Sho^Frequency> 
<KeyFrameLeveX> level-ld • . . <KeyFraiaeIpevttl> 
<Hi9hXightL«ngtli> lengrli . . . <HlgtaighU«ngth> 

</BrovrdingPr«ferences> 



20 
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30 



35 



The descriptor <BrowsingPreferences> specifies the browsing preferences of a user. The user's preferred 
views are specified by the descriptor < Views) . For each category, the preferred views are specif iect^by the descrip- 

40 tor (ViewCategory) with an id attribute which corresponds to the category id. The descriptor 
< FrameFrequency > specifies at what interval the frames should be displayed on a browsing slider under the frame 
view. The descriptor ( ShotFrequency ) specifies at what interval the shots should be displayed on a browsing slider 
under the shot view. The descriptor ( KeyFrameLevel) specifies at what level the key frames should be displayed 
on a browsing slider under the key frame view. The descriptor (HighlightLength) specifies which version of the 

45 highlight should be shown under the highlight view. 



so 
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RItering preferences 

<FilteringPreference3> 

<Cat:egorie3> category-name , • . </Categories> 
<ChanDels> channel -number . . . </Channels> 
<Itating5> rat lag- id - • . </Ra tings > 
<Show3> show- name . . . </Show3> 
<Author5> author-name . . . </Author3> 
<Producers> producer -name • . . </Produc«r3> 
<Director3> director-name • . . </Directors> 
<Actor3> actor-name . . - </Actora> 
<Reyword9> keyword . . . </Keyword3> 
<Title3> title-text ... </Titlc3> 
• • • ^ 

</ niter ingPreferences> 

The descriptor < FilteringPreferences > specrfies the filtering related preferences of a user. 
Search preferences 

<SearchPreference3> 

<Categories> category-name . - . </Categorie3> 

<Channels> channel-nimiber </Channels> 

, <Ratings> rating-id ... </Rating3> 
<Show3> 3ho*/-name . . . </Sho%^> 
cAuthors> author-name . . . </Author3> 

<Producer3> producer-name </Producers> 

<Director3> direct or -name </Oixectors> 

<Actor3> actor-name . . . </Actor3> 
<Keyword3> keyword". . . </Keyword3> 
<Titlea> title-text </Ticles> 

</SearchPreference3> 
The descriptor ( SearchPreferences) specrfies the search related preferences of a user. 
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Device preferences 

<I)evic«Prefer«nc«3> 

<Brlghcne39> bcightness-vaXu* </Bri9htness> 
<Contrast> cent raat -value </Contrast> 
<Volume> volume-value </Volume> 

</DevlcePr«£erttnces> 



The descriptor < DevicePreferences) specHies the device preferences of a user. 
15 Usage History 
[0055] 

Browsing history 



20 



25 



30 



35 



40 



<Brows ingHi 3 to ry> 
<View3> 

<ViewCategory id-**'> view- id - . . </ViewCategory> 
<ViewCategory id»*'*> view-id — </ViewCategory> 

</View3> - 
<rraiiieFrequency> frequency — <rramerrequency> 
<ShoCFrequency> frequency . . . <ShotFre<iuency> 
<KeyFraxneLevel> lev«l-ld • . - <KeyFranieLevel> 
'CHighligh^Leng^h> length . . . <aighiight:Length> 

</Brow3 ingH is tory> 



45 The descriptor ( BrowsingHistory > captures the history of a user's browvsing related activities. 

RItering history 
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<rilce ringHi3t:ory> 

<Categorie3> category-naiae </Categori«3> 
<Channttl3> channel*n\xxnber • - • </Chanxiel9> 
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<Rat:inga> rating-id — </Ilatiag3> 
<Show3> ahow^aame . . . </Sliow3> 
<Author3> author-name . . * </Attthor3> 
<?roduc«r3> producer-name . - - </Producer3> 
<0irector3> director-name — </Director3> 
<Actors> actor-naine • . . </Actor3> 
<Keyworda> keyword . . . </Keywords> 
<ritie3> title-text </TitXe3> 

</FilteringHi3tory> 

The descriptor < Filtering History > captures the history of a user's filterin^eiated activities. 
Search history 

<Sea rchHia tory> 

<Categorie3> category-name . . • </Categorie3> 
<Channels> chazmel-nuzober • . . </Cbannel3> 
<Rating3> rating-id ... </Rating3> 

<Show3> show-name </Show3> 

<Authors> author-name ... </Author3> 
<Producer3> producer-ziame ... </Producers> 
<Oirector3> director-name • • . </Director3> 
<ActQrs> actor-name ... </Actor5> 
<Keyword5> Jceyword </Keywordis> 
<Title3> title-text ... </Titlea> 

</Searchai3 tory> 

The descriptor < SearchHistory ) captures the history of a user's search related activities. 



.ioae8e7A2j_> 



29 



EP 1 026 887 A2 



Device history 

<t)eviceHl3tory> 

5 <Brightnc33> torightnesa-value — </Bri9htnes3> 

<Contxa3t> coatraat-vaioe </ContrMt> 

<Volume> volume-value . , . </Volxi»e> 

^0 </0eviceKt3tory> 



TTie descriptor ( DeviceHistory > captures the history of a user^s device related activities. 

IS 

User demographics 
[0056] 
20 • Age 

<Age> age </Age> 



25 

The descriptor ( Age) specifies the age of a user 
Gender 

30 <Gender> </G«Dd«r> 



35 



40 



45 



The descriptor (Gender) specifies the gender of a user. 
ZIP code 

<2IP> . </2IP> 

The descriptor ( ZIP ) specifies the ZIP code of where a user lives. 
System Description Scheme 

[0057] The proposed system description scheme includes four major sections for describing a user. The first sec- 
tion Identifies the described system. The second section keeps a list of all known users. The third section keeps lists of 
available programs. The fourth section describes the capabilities of the system. Therefore, the overall structure of the 
proposed description scheme is as follows: 



so 
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<?XML ver3ion-''1.0-> 

<!OOCTYPE MPEG-7 SYSTEM . dtd-'> 

<Sy3teniIdentity> 

<SysteiaIX» </SystemID> 

<Sy9texBName> • . . </SysteniMajDe> 

<Sy3temSeriallluinber> • . . </Sy3tesnS«rlalNumber> 
</ S ys temldent 1 ty> 
<Sy3 tiemD s e r s > 

<0ser3> . . , </0d«r3> 
</SystemU3er3> 
<Sy3temPrograi&3> 

<Cate9orle3> . . . </Ca€e9orle3> 

<ChanneX3> . . . < /Channel 3> 

<Progranis> . . - </Programs> 



</Sy3temProgram3> 
<Sy3tcntCapabiai.tltt3> 

<View3> . . , </View3> 
</Sy3teBCapabilitie3> 



40 



System Identity 
[0058] 

System ID 



45 



<SystemlD> system-id </Sy3temID> 



50 
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The descriptor < SystemID > contains a number or a string to identify a video system or device. 



System rame 



<Sy3teisNain«> systcm-naM </Sys^en]iHaM> 



The descriptor < SystemName) specifies the name of a video system or device. 



31 



BNSDOCID:<EP 10268e7A2 I > 



10 



15 



20 



25 



30 



35 



40 



EP 1 026 887 A2 

System serial number 

<SystemSerialNujDber> systvm-serial-aumbcr </Sy3tMS«rlalMuxziber> 

The descriptor ( SystemSerial Number > specifies ttie serial number of a video system or device- 
System Users 
[0059] 
Users 

<U3er3> 

<U3er> 

<U3erID> user-id </UserID> 
<Oserftamc> oaer-name </U5erNaae> 

</Uaer> 

<U3er> 

<U3erID> user-id </DserID> 
<03erl9aine> user-name </nserMaffle> 
</Cser> 

</03er3> 



The descriptor < SystemUsers ) lists a number of users who have registered on a video system or device. Each 
user is specified by the descriptor ( User) . The descriptor ( User ID ) specifies a number or a string which should 
match with the number or string specified in < UserlD ) in one of the user description schemes. 



Programs In the System 
[0060] 
45 • Categories 



<Cate9ories> 

so 
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<Cat:egory> 

<CategoryID> category-id </CategoryID> 
<CategoryNaiDe> category-najae </Cate9oryttaiDft> 
<SubCattt9orles> 3\2b*category-id ... </SuJbCat:egories> 

</Category> 

<Category> 

<CategoryID> category- id </CategoryID> 
<CategoryMazae> cacegory-naxoe </CategoryNanie> 
<SiibCategorles> aub-category-id </SubCategorlas> 

</Category> 

</Categories> 

The descriptor < Categories) lists a number of categories which have been registered on a video system or 
device. Each category is specified by the descriptor (Category). The major-sub relationship between categories 
is captured by the descriptor < Subcategories) . 

Channels 

<ChanneIs> 

<Channel> 

<ChannelXD> channel-Id </ChanneIID> 
<ChannelHaiite> channel-name </ChannelJlame> 
<SubChannela> 312b- channel-id • . « </SubChanncla> 

</ChaAnel> 

<Chaxmel> 

<ChannelID> chaxmel-id </ChannelIO> 
<Chaxinelllaine> channel-name </ChannUNaBie> 
<SubChazmels> aub-channel-id ... </Sul>Channels> 



</Channel> 



</Channela> 
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TTie descriptor (Channels) lists a number o1 channels which have been registered on a video system or 
device. Each channel is specified by the deso-iptor (Channel). The major-sub relationshp between channels is 
captured by the descriptor ( Subchannels) . 

Programs 

<Pro9rain3> 

<Ca tego ryPrograms> 

<CatcgoryID> category- id </CategoryID> 

<Brogr9ms> program-id * . . </Prograiiis> 
< /Ca tego ry Prog rams > 
<CategoryPro9ra2ns> 

<CategoryXD> category^id </CategoryID> 

<Prograitts> program- id - - . </Prograffi3> 
</Category Prog rams > ^ 

<CtianixelProgram3> 

<ChaiuielID> channel-id </ChannelIO> 

<Progxams> program- id ^ </Programs> 
</ChanneXPrograin3> 
<CbanaeJlProgram3> 

<ChaimelIO> channel-id </QiannelXO> 

<Progzams> program- id ... </Programs> 
< /ChanaelFro9raffls> 

</Prograina> 

The descriptor ( Programs) lists programs who are available on a video system or device. The programs are 
grouped under corresporxJing categories or channels. Each group of progran^s are specified by the descriptor 
(CategoryPrograms) or ( ChannelPrograms ) . Each program id contained in the descriptor (Programs) should 
match with the number or string specified in ( ProgramID) in one of the program description schemes. 
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System Capabilities 
[0061] 
Views 



25 



<View3> 

<View> 

<ViewID> view-Id </VicwID> 
<ViewNaxae> view-name </Vie%#Naine> 

</View> 

<View> 

<ViewrD> view-id </ViewID> 
<ViewNasie> view-narae </ViewKa]De> 
</Vicw> ^ 

</ViCW3> 



The descriptor { Views > lists views wtiicli are supported by a video system or device. Each view is specified by 
30 the descriptor ( View > . The descriptor < ViewName > contains a string which should match with one of the following 
views used In the program description schemes: ThumbnailView, SlideView. FrameView, ShotView, Key- 
FrameView. HighlightView. EventView, and CloseUp View. 

[0062] The present inventors came to the realization that the program description scheme may be further modified 

35 to provide additional capabilities. Referring to FIG. 1 3. the modified program description scheme 400 includes four sep- 
arate types of information, namely a syntactic structure description scheme 402, a semantic structure desaiption 
scheme 404, a visualization description scheme 406, and a meta information description scheme 408. It is to be under- 
stood that in any particular system one or more of the description schemes may be included, as desired. 
[0063] Referring to FIG. 14, the visualization description scheme 406 enal^les last and effective browsing of video 

40 program (and audio programs) by allowing access to the necessary data, preferatrfy in a one-step process. The visual- 
ization description scheme 406 provides for several different presentations of the video content (or audio), such as for 
example, a thumbnail view description scheme 41 0. a key frame view description scheme 41 2, a highlight view desmp- 
tion scheme 414, an event view description scheme 416, a dose-tp view description scheme 418. and an alternative 
view description scheme 420. Other presentation techniques and description schemes may be added, as desired. The 

45 thumbnail view description scheme 410 preferably includes an image 422 or reference to an image representative of 
the video content and a time reference 424 to the video. The key frame view description scheme 412 preferably includes 
a level indicator 426 and a time reference 428. The level irxiicator 426 accommodates the presentation of a different 
number of key frames for the same video portion deperxiing on the user's preferoice. The highlight view description 
scheme 414 includes a length indicator 430 and a time reference 432. The length indicator 430 accomnxxJates the 

so presentation of a different highlight duration of a video depending on the user's prefer erx:e. The event view description 
scheme 416 preferably includes an event indicator 434 for the selection of the desired event arxl a time reference 436. 
The close-up view descriptfon scheme 418 preferat>ly includes a target indicator 438 and a time reference 440. The 
alternate view description scheme preferably includes a source indicator 442. To increase performance of the system it 
is preferred to specify the data which is needed to render such views in a centralized arxJ straightforward manner. By 

55 doing so, it is then feasible to access the data in a simple one-step process witfxjut complex parsing of the video. 
[0064] Refening to FIG. 15, the mgta information description scheme 408 generally includes various descriptors 
which carry general information about a vkJeo (or audio) program such as the title, category, keywords, etc. Additional 
descriptors, such as those previously descrft>ed, may be included, as desired. 
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[0065] Referring again to FIG. 13, the syntactic structure description scheme 402 specifies the physical structure 
of a video program (or audio), e.g.. a table of contents. The physical features, may include for example, color, texture, 
motion, etc. The syntactic structure description scheme 402 preferably includes three modules, namely a segment 
description scheme 450. a region description scheme 452, and a segment/region relation graph description scheme 

5 454. The segment description scheme 450 may be used to define relationships between different portions of the video 
consisting of multiple frames of the video. A segment description scheme 450 may contain another segment description 
scheme 450 and/or shot description scheme to form a segment tree. Such a segment tree may be used to define a tem- 
poral structure of a video program. Multiple segment trees may be created and thereby create multiple table of contents. 
For example, a video program may be segmented into story units, scenes, and shots, from which the segment descrip- 

70 tion scheme 450 may contain such information as a table of contents. The shot description scheme may contain a 
number of key frame description schemes, a mosaic description scheme(s), a camera motion description scheme(s), 
etc. The Key frame description scheme may contain a still image description scheme which may in turn contains color 
arxJ texture descriptors. It is noted that various low level desaiptors may be included in the still image description 
scheme under the segment description scheme. Also, the visual descriptors may be included in the region description 

15 scheme which is not necessarily under a still image description scheme. On example of a segment description scheme 
450 is shown in FIG. 16. 

[0066] Referring to FIG. 17, the region description scheme 452 defines the interrelationships between groups of 
pixels of the same andJor different frames of the video. The region description scheme 452 may also contain geometri- 
cal features, color, texture features, motion features, etc. 

20 [0067] Referring to FIG. 18, the segment/region relation graph description scheme 454 defines the interrelation- 
ships between a plurality of regions (or region description schemes), a plurality of segments (or segment description 
schemes), and/or a plurality of regions (or description schemes) and segments (or description schemes). 
[0068] Refening again to FIG. 13, the semantic structure description scheme 404 is used to specify semantic fea- 
tures of a video program (or audio), e.g. semantic events. In a similar manner to the syntactic structure description 

25 scheme, the semantic structure desCTiption scheme 404 preferably includes three modules, namely an event descrip- 
tion scheme 480, an object description scheme 482, and an event/objection relation graph description scheme 484. 
The event description scheme 480 may be used to form relationships between different events of the video normally 
consisting of multiple frames of the video. An event description scheme 480 may contain another event description 
scheme 480 to form a segment tree. Such an event segment tree may be used to define a semantic index table for a 

30 video program. Multiple event trees may be created and tiiereby creating multiple index tables. For exanple. a video 
program may include multiple events, such as a basketball dunk, a fast break, and a free throw, and the event descrip- 
tion scheme nrtay contain such irrformation as an index table. The event description scheme nrtay also contain refer- 
ences which link the event to the corresponding segments and/or regions specified in the syntactic structure description 
scheme. On exanple of an event description scheme is shown in FIG. 19. 

35 [0069] Refening to FIG. 20. the object description scheme 482 defines the interrelationships between groups of 
pixels of the same and/or different frames of the video representative of objects. TTie object description scheme 482 
may contain another object desaiption scheme and therdDy form an object tree. Such an object tree may be used to 
define an object index table for a video program. The object description scheme may also contain references which link 
the object to the corresponding segments arKl/or regions specified in the syntactic structure descrptiott scheme. 

40 [0070] Referring to FIG. 21, the event/object relation graph description scheme 484 defines the interrelationships 
between a plurality of events (or event description schemes), a plurality of objects (or object description schemes), 
arxl/or a plurality of events (or description schemes) and objects (or descriptbn schemes). 

[0071] The terms and expressions that have been employed in the foregoing specification are sued as terms of 
description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equiv- 
45 alents of the features shown and described or portions thereof, it being recognized that the scope of the invention is 
defined and limited only by the claims that follow. 

Clainns 

so 1 . A method of using a system (12) with at least one of audio, an image, and a video comprising a plurality of frames 
comprising the steps of: 

(a) providing at least one of the following: 

55 (i) a program description scheme (18) containing information (10) related to at least one of information 

regarding interrelationships between a plurality of said frames, characteristics of the content of a plurality 
of said frames, characteristics of the content of said audio, characteristics of the content of said image, 
characteristics of the content of said video; 
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(ii) a user description scheme (20) containing information (14) related to at least one of a user's personal 
preferences, information related to said user, a user's viewing history, and a user's listening history; 
(lii) a system description scheme (22) containing information (10) regarding at least one of available vid- 
eos, available categories, available channels, available users, available images, capabilities of a device for 
5 providing said at least one of said audio, said image, and said video to a user, relationship between at least 

two of said video, said program description scheme (18), and said user description scheme (20). relation- 
ship between at least two of said audio, said program description scheme (18), and said user description 
scheme (20). relationship between at least two of said image, said program description scheme (18). and 
said user description scheme (20); and 

70 

(b) selecting at least one of a video, an image, and audio based upon said at least one of said program descrip- 
tion scheme (18). said user description scheme (20), and said system description scheme (22). 

2. A method of using a system (1 2) with at least one of audio, image, and a video comprising a plurality of frames com- 
75 prising the steps of: 

(a) providing at least two of the following: 

(i) a program description scheme (18) containing information (10) related to at least one of information 
20 regarding interrelationships between a plurality of said frames, characteristics of the content of a plurality 

of said frames, characteristics of the content of said audio, characteristics of the content of said image, 
characteristics of the content of said video; * 

(ii) a user description scheme (20) containing information (14) related to at least one of a user's personal 
preferences, infcwmation related to said user, a user*s viewing history, and a user's listening history; 

25 (iii) a system description scheme (22) containing information (10) regarding at least one of availafc>le vid- 

eos, available categories, available channels, available users, available images, capabilities of a device for 
providing said at least one of said audio, said image, and said video to a user, relationship between at least 
two of said video, said program description scheme (18). and said user description scheme (20)/ relation- 
ship between at least two of said audio, said program description scheme (18), and said user description 

30 scheme (20), relationship between at least two of said image, said program descrption scheme (18), and 

said user description scheme (20); and 

(b) selecting at least one of a video, an image, and audio based upon said at least two of said program descrip- 
tion scheme (18), said user description scheme (20). and said system description scheme (22). 

35 

3. The method of claim 1 or 2 wherein said program description scheme (18) includes at least one of a title, a cate- 
gory, an annotation, a keyword, and a date related to said plurality of said frames. 

4. The method of claim 1 or 2 wherein said program description scheme (1 8) includes fields for storing (1 ) Information 
40 regarding interrelationships between said plurality of said frames includes the identification of key frames of said 

- video, (2) Information regarding interrelationships between said plurality of said frames includes the identification 
of a plurality of said frames representative of the highlights of at least a portion of said video, (3) information regard- 
ing interrelationships between said plurality of said frames includes the identification of a set of frames, each of 
. :v which is representative of a different portion of said video. (4) and information regarding interrelationships between 
45 said plurality of said frames includes the identification of a plurality of sequential frames of said video that represent 
at least one of a shot arKi a scene. 

5. The method of daim 1 or 2 w^herein said program description scheme (1 8) includes f iekfe for storing at least one of 
a color profile of at least a portion of said video, a texture profile of at least a portion of said video, a shape profile 

so of at least a portion of said video, and a motion profile of at least a portion of said video. 

6- The method of daim 1 or 2 wherein said program description scheme (18) identifies a second audio track separate 
from the normal audio track of said video. 

55 7. The metfTod of claim 1 or 2 wherein said program description scheme (18) identifies Internet based information 
related to said video. 

8. The method of daim 1 or 2 wherein said program description scheme (18) includes syntactic structure of a plurality 
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of frames of said vkjeo. 

9. The method of daim 1 or 2 wherein said program description scheme (18) includes semantic structure regarding a 
plurality of said frames of said video. 

5 

1 0. The method of daim 1 or 2 wherein information for said program description scheme (1 8) is extracted from the con- 
tent of a video itself. 

1 1 . The method of daim 1 or 2 further comprising the step of generating a summary of said video of a user determined 
10 duration t>ased upon said information of said program description scheme (18). 

12. The method of claim 1 or 2 further comprisir>g the steps of: 

(a) generating at least one of summary and key frame information of said video based upon the content of said 
15 video; and 

(b) Including said at least one of said summary and said key frame irrformation in said program description 
scheme (18). 

13. The method of claim 1 or 2 wherein said user description scheme (20) contains Information related to said user's 
20 viewing history. 

14. The method of claim 1 or 2 wherein said information (14) related to said user includes at least one of geographic 
infornnation arxJ demographic information. 

25 15. The method of claim 1 or 2 wherein said user description scheme (20) contains a user's personal preferences. 

1 6. The method of claim 1 or 2 wherein said user description scheme (20) is contained in a handheld electronic device. 

1 7. The method of claim 1 or 2 wherein said user description scheme (20) contains at least one of preselected f requen- 
30 cies and preselected stations for radio broadcasts. 

18. The method of claim 1 or 2 wherein said system description scheme (22) includes capabilities of said device (12) 
for providing said at least one of said audio, said image, and said video to said user. 

35 1 9. The method of claim 1 or 2 wherein said video is received from at least one of broadcast televlsbn, cable television, 
satellite televisionp digital television. Internet broadcasts, world wide web. digital video discs, still images, video 
cameras, laser discs, magnetic media, conputer hard drive, video tape, data services, and microwave communi- 
cations, 

40 20. The method of claim 1 or 2 wherein said program description scheme (18) is received from at least one of 
PSIP/DVB-SI information in digital television broadcasts, spedalized digital television data services, spedalized 
Internet services, data file, data over a telephone wire, and computer menrK)ry. 

21. The method of claim 1 or 2 further comprising the step of in response to receiving said video determining together 
45 with information witiiin said user description scheme (20) whether to perform an analysis of the content of said 

video. 

22. The method of claim 1 or 2 further comprising the steps of: 

so (a) extracting information contained in at least one of the system description scheme (22) and tiie program 

description scheme (18): and 

(b) nx)difying said information (14) contained in said user description scheme (20) based on upon said extract- 
ing. 

55 23. The method of claim 1 or 2 further comprising the steps of: 

(a) storing said user description scheme (20) on a first portak>le device; 

(b) interconnecting said portat»le device with a plurality of different second devices, each of which uses tiie 
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irrformation (14) contained within said user description scheme (20). 

24. The method of daim 1 or 2 wherein said program description scheme (18) is included within a camera and the sys- 
tem modifies said information contained within said camera based on, at least in part, said information of said user 

5 description scheme (20) and said information of said system desaiption scheme (22). 

25. The method of claim 1 or 2 further comprising a search device to identify video based on. at least In part, said Infor- 
mation (10) of said program description scheme (18) and said information (14) of said user description scheme 
(20). 

10 

26. The method of daim 1 or 2 further comprising a recording device ttiat records video broadcasts that may be desir- 
able to said user t>ased upon said information (10) contained within said program description scheme (18) and said 
information (14) contained within said user description scheme (20). 

75 27. The method of claim 1 or 2 further comprising the steps of: 

(a) storing at least one of said user description scheme (20), said system description scheme (22) and said 
program description scheme (18) on a first device; and 

(b) transferring at least one of said user description scheme (20), said system description scheme (22) and 
20 said program description scheme (18) to a second device through a network. 

28. The method of claim 1 or 2 further comprising the steps of: * 

(a) providing the system description scheme (22) across a network to a provider of at least one of audio, image. 
25 and video data; 

(b) in response to receiving said system description scheme (22) said provider selecting said at least one of 
audio, image, and video data in accordance with said system description scheme (22); and 

(c) providing said at least one of said audio, image and video data to a device (12) for a user. 

30 
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